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What We Studied 


Post-secondary education has become a gateway for economic and social mobility in U.S. society. While both federal 
and state policy has sought to broaden access to higher education, there remain substantial obstacles to expanding the 
more crucial objective of student degree completion. One challenge is that many students exit high school under- 
prepared for college-level work—particularly in mathematics. For instance, as of 2010, over half of students entering 
community colleges in the U.S. were deemed to be unprepared for college-level course-work in mathematics, while 
one third of students were deemed unprepared for college-level reading.! Students who are under-prepared for college- 
level course work are often referred to or required to complete developmental courses, which are intended to help them 
learn skills necessary to do well in college-level work. Large numbers of students begin such courses, and low-income 
students and minority students are disproportionately likely to take developmental coursework.” Although 
developmental courses are intended to help students succeed, research has suggested that they may actually have the 
effect of raising barriers to success in college, hindering credit and degree completion.*> Evidence from a range of 
settings—including 2- and 4-year public institutions in Texas—indicates that participation in conventional 
developmental coursework does little to improve student persistence and credit completion, relative to immediately 
beginning college-level coursework.>* 


To address under-preparation of students for college-level course-work, Texas House Bill 5 (HBS) introduced a 
number of substantive changes to the state high school curriculum and graduation requirements. Among its provisions, 
HBS created a requirement that school districts offer a college preparatory mathematics course for students not meeting 
college readiness standards in mathematics by the end of their third year of high school. It further required that the 
course be offered through a partnership with an institution of higher education—typically a community college—and 
that successful completion of the course must satisfy the partner institution’s requirements for enrollment in college- 
level coursework. 


One curriculum designed with the goals and requirements of HBS is the Transition to College Mathematics Course 
(TCMC), developed by the Charles A. Dana Center. The Dana center developed TCMC as a model college preparatory 
math course, melding previously developed secondary-level course materials with strategies they had used to build 
college-level developmental courses. TCMC differs from conventional remedial math courses in several respects. First, 
the course content aligns with the multiple mathematics pathways framework®'° adopted by many Texas higher 
education institutions, providing a coherent sequence of work across the transition from high school to higher 
education. Second, the course involves novel material and instructional strategies, rather than repetition of content that 
students have already encountered. Third, the course incorporates evidence-based pedagogical approaches including 
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putting greater emphasis on richly contextualized applications, developing students’ self-regulated learning strategies 
and productive persistence, and varying instructional activities. Taken together, these differences provide reason to 
expect that student participation in TCMC could have immediate and longer-term impacts on student outcomes. 

In this project, we sought to evaluate the impacts of TCMC in its first and second years of implementation. Our 
guiding research question was: relative to taking typical high school coursework, what are the effects of participating 
in TCMC on high school graduation, post-secondary enrollment, and progress in college-level mathematics for twelfth 
grade students enrolled in TCMC? 


How We Analyzed the Data 


TCMC was offered at high schools in eight districts across central Texas during the 2016-17 school year and in thirty 
districts during the 2017-18 school year. Within high schools that offered the course, enrollment in TCMC was at the 
discretion of students and school staff. Most students who enrolled in the course were in twelfth grade. The main rule 
guiding enrollment was that students should not be college-ready at the beginning of twelfth grade. Schools could use a 
variety of approaches for determining college readiness, including college readiness tests, other standardized test 
scores, or grades. Schools did not follow any consistent approach for determining college readiness or which students 
should take the course, and advising practices differed from school to school. 


To estimate the effects of participating in the course, we compared twelfth grade students who participated in TCMC to 
observationally similar twelfth grade students from the same cohort, but who did not enroll in the course.* 

We aimed to create groups that were closely matched on background characteristics that may have influenced students 
to enroll in the course and that may be associated with later student outcomes. Using data from the Texas PK-20 
Workforce Database, we identified matched comparison groups based on students’ contemporaneous demographic and 
program enrollment status (e.g., Special Education, gifted programs), demographic history and past program 
enrollments, math course-taking history, and scores on the STAAR Algebra I end-of-course exam. Appendix Table S1 
lists the full set of covariates used. We created the comparison groups using inverse propensity score weighting 
methods, with propensity scores estimated by generalized boosted regression models, a machine learning technique 
that is suitable for use with large sets of covariates.'”'> After weighting based on propensity scores, the set of students 
in the comparison group was very similar to the set of students who took TCMC in terms of demographic 
characteristics, academic performance, and past math coursework.+ 


To the extent that these background characteristics explain whether a student enrolled in TCMC, differences in 
outcomes between students who took TCMC and students in the comparison group can be attributed to the impact of 
the program. We assessed differences between TCMC participants and comparison students on outcomes related to 
students’ post-secondary success. The main goal of the TCMC program was to improve student preparedness for 
college-level math. Thus, the primary outcomes of interest were enrollment and passage rates for college-level math 
courses. For completeness, we also examined enrollment and passage rates for developmental math courses at the post- 
secondary level. In order to affect post-secondary performance outcomes, however, students must first graduate and 
enroll in post-secondary education. Therefore, we also examined impacts on high school graduation and college 
enrollment rates as intermediate outcomes, including dis-aggregated college enrollment rates in community colleges, 
and public four-year colleges or universities. For all of the college-level outcomes, we examined cumulative rates for 
four semesters after the students’ high school year (i.e., Fall, Spring, Summer, Fall). 


To generate average effect estimates, we regressed each outcome on the covariates (listed in Table $1), indicators for 
each school, covariate-by-treatment interactions, and school-by-treatment interactions. Regressions used inverse 
propensity score weights. We then calculated a weighted average of the school-specific estimates, with weights based 
on the size of the TCMC group in each school. 


* For the 2016-17 cohort, we also made comparisons between students who participated in TCMC and observationally similar 
students who were in twelfth grade during the 2015-16 school year, prior to when TCMC was offered.!! This approach was not 
possible with the 2017-18 cohort because a substantial portion of the second cohort was implementing TCMC for a second year. 
? Appendix Figure S1 depicts the degree of similarity (or “balance”) between the TCMC students and comparison students from 
the 2016-17 cohort. Appendix Figures $2 through S4 show the degree of balance for the 2017-18 cohort. 
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What We Discovered 


In this brief, we focus on outcomes for the Fall semester one and two years after enrolling in the course. Table 1 
reports rates for key outcomes among students who participated in TCMC and among students in the comparison 
group, in each of the two student cohorts. For each cohort, the columns labeled “Difference” are the estimated 
difference between the TCMC group and the comparison group, which represent our estimates of the effects of 
participating in TCMC. 


Across both cohorts, students who participated in TCMC graduated from high school at higher rates than students in 
the comparison groups, with differences in on-time graduation rates of 3.1 percentage points (SE = 0.9) for the 2016-17 
cohort and 4.7 percentage points (SE = 0.4) for the 2017-18 cohort. However, overall rates of college enrollment 
followed more complex and ambiguous pattern. For the 2016-17 cohort, TCMC students enrolled in college at 
somewhat lower rates than comparison students (43.7% versus 47.2% by Fall semester of Year 2). In the 2017-18 
cohort, overall rates of college enrollment were similar, with a difference of 0.6 percentage points that was not 
statistically distinguishable from zero. 


Possible differences in college enrollment rates appear to have been driven by reductions in enrollment at four-year 
colleges and universities and, potentially, partially or fully offsetting increases in enrollment at community colleges. 
By two years after start of their senior year, TCMC students in the 2016-17 cohort enrolled at four-year institutions at a 
rate 4.5 percentage points lower (SE = 1.3) than students in the matched comparison group of the same cohort, offset 
by a difference of 1.4 percentage points (SE = 1.8) in community college enrollment. For the 2017-18 cohort, the 
difference in enrollment rates at four-year institutions was smaller in magnitude (8.8% among TCMC students versus 
11.3% among 

comparison students, a 


difference of -2.5 Table 1. Estimated average outcomes of TCMC students and comparison students for high school graduation, 
percentage p oints. SE = post-secondary enrollment, math course enrollment, and math course passage. 


0.7) and the difference 2016-17 Cohort 2017-18 Cohort 
in community college Outcome TCMC fom roa Difference (SE) p-value ‘ae ee Difference (SE) p-value 
enrollment rates was 1,066) ESS = 2,452) 2468) ESS=7.971) 
larger and statistically “High schoolgraduation(%) #2 2 
distinguishable from fearl 95.5 92.5 3.1 (0.9) <.001 98.7 94.0 47 (0.4) <.001 
zero (34.9% among Picante ae dal a 2.8 (0.8) .001 99.2 95.4 3.8 (0.3) <.001 

Any post-secondary enrollment (% 
TCMC students versus Year1 33.5 37.6 41 (1.8) .020 36.4 35.8 0.6 (1.2) .603 
30.9% among Year2 43.7 47.2 -3.4 (1.9) .063 42.8 40.8 2.0 (1.2) .093 
comparison students, a Post-secondary enrollment at community college (%) 

. Yearl 21.6 19.8 18 (1.6) .257 27.9 24.6 3.2 (1.1) .004 
difference of 4.0 Year2 340 32.6 14 (1.8) 441 34.9 30.9 4.0 (1.2) <.001 
percentage points, SE= Post-secondary enrollment at partner community college (%) 

1.2). Yearl 116 11.8 -0.1 (1.2) 916 213 19.4 1.9 (1.0) .055 

Year2 178 18.5 -0.7 (1.4) .604 26.9 24.8 2.0 (1.1) .059 
Participating in TCMC Post-secondary enrollment at 4-year public college or university (%) 

Year1 10.5 15.9 5.4 (1.2) <.001 7.2 9.7 -2.5 (0.6) <.001 
may have reduced Year2 128 173 45 (13) <001 88 113 25 (0.7) <001 
colle ge-level math College-level math course enrollment (%) 
course enrollment rates Yearl 147 17.2 -2.5 (1.4) .066 44 6.0 -1.7 (0.5) .001 

: Year2 21.6 24.8 3.2 (1.6) .037 12.3 14.8 2.4 (0.8) .003 
and passing rates. For College-level math course passage (%) 
both cohorts, students Yearl 8.0 12.5 -4.5 (1.1) <.001 3.0 45 -1.6 (0.4) <.001 
who enrolled in TCMC Year2 13.5 19.5 -6.0 (1.3) <.001 7.6 10.0 -2.5 (0.7) <.001 

Developmental math course enrollment (%) 

were somewhat less Yearl 95 11.0 -14 (1.2) .230 11 12 0.2 (0.3) 52 
likely than students in Year2 128 14.7 -1.8 (1.4) .178 3.5 3.4 0.2 (0.5) .695 
the comparison groups Developmental math course passage (%) 

: Yearl 43 5.7 -1.5 (0.9) .091 0.7 0.7 -0.1 (0.2) 805 
to enroll in a college- Year2 62 87 25 0) 015 ; 2 


level math course by 

the Fall semester of the Notes: Difference = difference in percentage points between students who participated in TCMC and those that did not enroll. SE 
= Standard error of the difference. P-value = p-value for the null hypothesis of no difference between TCMC and comparison 
groups. N = total sample size of students. ESS = effective sample size, based on inverse propensity weights. 
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second year after high school graduation, with differences of -3.2 percentage points (SE = 1.6) for the 2016-17 cohort 
and -2.4 percentage points (SE = 0.8) for the 2017-18 cohort. Similarly, TCMC students were less likely than students 
in the comparison groups to have completed a college-level math course by the Fall of the second year after high 
school graduation. Regarding developmental coursework, TCMC enrollees and comparison students enrolled in 
developmental math courses at similar rates. 


Policy Recommendations 


Findings from this study suggest that participating in TCMC could have shifted some students who might otherwise 
have enrolled in four-year institutions towards enrolling in community college instead. This impact would be 
consistent with the incentive structure of the college preparatory course requirements created by HB 5, in that the 
successful completion of the course provided exemption from proficiency exams only at the partner institution. 
However, it is also possible that the pattern of results could be due to pre-existing differences between TCMC students 
and comparison students—particularly differences in aspirations or goals for attending four-year colleges—for which 
our analysis was not able to adjust. 


Given the possibility of pre-existing differences between TCMC participants and comparison students, we urge caution 
in drawing any implications regarding the causal impact of participating in TCMC. The differences observed here 
could have resulted from our inability to fully adjust for initial differences in college readiness, college aspirations, or 
other potential confounders, at the start of students' senior year. Of particular concern is the possibility that the 
comparison group could have included some students who had already achieved college readiness by the start of senior 
year. Relative to students in TCMC, college-ready students in the comparison group could be expected to be more 
likely to apply to and gain admission into four-year colleges, which might explain the differences in four-year 
enrollment rates that we observed. In order to more effectively evaluate TCMC—and other similar programs—it will 
be critical to consider and account for students’ college readiness status while students are still in high schools. 
Incorporating such data into the Texas PK-20 Workforce Database would enhance the capacity of the ERC research 
community for understanding the impacts of policy changes, programs, and other strategies aimed at improving student 
success across the transition between secondary and post-secondary education. 


A final policy recommendation is to continue investigating the implementation of TCMC and other college preparatory 
math courses implemented in response to HBS. Further work should investigate how schools assess college readiness 
and determine whether students should take college readiness courses. It would also be valuable to investigate 
students’ perspectives and experiences in such courses as they navigate the transition from high school to college. 


Note: At the time that this project was conducted, Pustejovsky was assistant professor in the Educational Psychology 
Department at the University of Texas at Austin. 
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Appendix 
Additional Tables and Figures 


At risk for dropping 
out 


Giftedness 


Immigrant status 


Special education 
status 


Limited English 
proficiency 


Prior math course- 
taking 


Prior math 
performance 


History of economic 
disadvantage, at-risk 
for dropping out, 
giftedness, immigrant 
status, and special 
education status 


Table S1 
Covariate definitions 
Variable Definition 
Sex Sex: male/female. 
Race/Ethnicity Race/ethnicity: Asian American, African American, Hispanic, American Indian, Pacific Islander, 
Multiracial, and White. 
Economic Economic disadvantage status for the 12™ grade year: free lunch status, reduced lunch status, no 
disadvantage disadvantage or other disadvantage. 


Indicator for whether a student was at risk for dropping out of school according to state-defined criteria 
as of the beginning of the 12th grade year. 


Indicator for whether a student participated in state-approved gifted and talented program during the 
12" grade year. 


Indicator for whether a student was identified as an immigrant according to the definition in Title II of 
No Child Left Behind Act of 2001, as of the beginning of the 12th grade year. 


Indicator for whether a student participated in special education instructional and related services 
program or general education program using special education services, supplementary aids, or other 
special arrangements for the 12th grade year. 


Indicator for whether a student was limited English proficient as determined by Language Proficiency 
Assessment Committee (LPAC) as of the end of the 12th grade year. 


Indicators for whether a student passed, failed, or did not take Grade 8 Mathematics (four years prior), 
Algebra I (three and four years prior), Geometry (two and three years prior), Algebra I (one and two 
years prior) and Pre-calculus (one year prior). 


STAAR end-of-course exam score for Algebra I, completed between 2014 and 2017. 


We tracked these variables from 2010 through 2017. Variables include (1) the number of years of 
available tracked data; (2) the number of years that a student was indicated as being in any of the 
categories for economic disadvantage and the number of years that the student was indicated as being in 
special education program, in gifted program, an immigrant, and at risk; (3) the proportion of years (the 
number of years the student was in the category divided by the number of years of record available) for 
the economic disadvantages categories; and, (4) if the student was ever indicated as being in special 
education program, in gifted program, an immigrant, and at risk. 
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Figure S1. Balance between TCMC group and comparison group in the 2016-17 cohort, before and after inverse 


propensity score re-weighting adjustment. 
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Figure $2. Balance between TCMC group and comparison group in the 2017-18 cohort on demographic 
characteristics and prior academic performance, before and after inverse propensity score re-weighting 


adjustment. 
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Figure S3. Balance between TCMC group and comparison group in the 2017-18 cohort on math course-taking 
patterns, before and after inverse propensity score re-weighting adjustment. 
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Figure S4. Balance between TCMC group and comparison group in the 2017-18 cohort on dual-credit and AP 
math course-taking patterns, before and after inverse propensity score re-weighting adjustment. 
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